Skip to content

Upgrade spiceai-52 to Vortex 0.68.0#42

Merged
lukekim merged 447 commits into
spiceai-52from
lukim/spiceai-52-vortex-0.68.0
Apr 10, 2026
Merged

Upgrade spiceai-52 to Vortex 0.68.0#42
lukekim merged 447 commits into
spiceai-52from
lukim/spiceai-52-vortex-0.68.0

Conversation

@lukekim
Copy link
Copy Markdown

@lukekim lukekim commented Apr 10, 2026

Summary

  • sync spiceai-52 to upstream Vortex 0.68.0 (ae906c7bfebb830f73401f3d5f8fcdeb9c5fc033)
  • preserve SpiceAI-specific DF52 compatibility and custom date->timestamp extension casting during the merge
  • regenerate public API lockfiles after the sync

Validation

  • RUSTC_WRAPPER= cargo test --no-run -p vortex-array -p vortex-datafusion
  • RUSTC_WRAPPER= cargo test -p vortex-array cast_date_days_to_timestamp -- --nocapture
  • RUSTC_WRAPPER= cargo test -p vortex-array test_map_array_conversion -- --nocapture
  • RUSTC_WRAPPER= cargo test -p vortex-datafusion case_when -- --nocapture
  • cargo +nightly fmt --all
  • ./scripts/public-api.sh
  • RUSTC_WRAPPER= cargo clippy --all-targets --all-features

asubiotto and others added 30 commits March 18, 2026 13:28
## Summary

This previous code was short-circuiting with a zero offset, with the
assumption that ends were already correctly physically sliced. However,
slicing is a lazy operation using metadata, so this commit fixes a
couple of edge cases by executing the fast path only if the ends are
correcly sliced. Otherwise, ends are trimmed and transformed with the
offset in for correct arrow array construction.

<!--
Thank you for submitting a pull request! We appreciate your time and
effort.

Please make sure to provide enough information so that we can review
your pull
request. The Summary and Testing sections below contain guidance on what
to
include.
-->

<!--
If this PR is related to a tracked effort, please link to the relevant
issue
here (e.g., `Closes: vortex-data#123`). Otherwise, feel free to ignore / delete
this.

In this section, please:

1. Explain the rationale for this change.
2. Summarize the changes included in this PR.

A general rule of thumb is that larger PRs should have larger summaries.
If
there are a lot of changes, please help us review the code by explaining
what
was changed and why.

If there is an issue or discussion attached, there is no need to
duplicate all
the details, but clarity is always preferred over brevity.
-->

<!--
## API Changes

Uncomment this section if there are any user-facing changes.

Consider whether the change affects users in one of the following ways:

1. Breaks public APIs in some way.
2. Changes the underlying behavior of one of the engine integrations.
3. Should some documentation be updated to reflect this change?

If a public API is changed in a breaking manner, make sure to add the
appropriate label. You can run `./scripts/public-api.sh` locally to see
if there
are any public API changes (and this also runs in our CI).
-->

## Testing

<!--
Please describe how this change was tested. Here are some common
categories for
testing in Vortex:

1. Verifying existing behavior is maintained.
2. Verifying new behavior and functionality works correctly.
3. Serialization compatibility (backwards and forwards) should be
maintained or
   explicitly broken.
-->
Enhanced tests to reproduce e2e arrow execution error and added some
unit tests to build_run_array

Signed-off-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com>
Remove #[inline] from private, pub(crate), and inner functions where the compiler already has full visibility, and from large (>20 line) functions where code bloat outweighs benefit.

Add #[inline] to BitBuffer operator trait impls (BitOr, BitAnd, Not, BitXor) — non-generic impls on a concrete type that are otherwise opaque cross-crate with LTO disabled.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…ta#7013)

Uses a new prebuild image with deps changed around.

I will remove the v2 after merging and simplify

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This PR includes initial support for the Variant DType, as described in
the [Variant
RFC](https://github.com/vortex-data/rfcs/blob/develop/accepted/0015-variant-type.md).
It includes most of the required boilerplate and initial structure for
this new dtype.

It includes the following changes:
1. New dtype
2. serialization for the dtype
3. Scalar variant and new scalar value
4. A lot of code paths that aren't supported yet and error accordingly.

---------

Signed-off-by: Adam Gutglick <adam@spiraldb.com>
…rtex-data#7014)

After vortex-data#6910, Older files with extension types will have zone stats that
are just based on the storage type (which might be arbitrary) and which
is rarely useful. we currently don't handle its existence at all, so
reading these files might panic or otherwise error. This PR falls back
to the storage type on the read path, but might not be the desirable
solution.

Signed-off-by: Adam Gutglick <adam@spiraldb.com>
<!--
Thank you for submitting a pull request! We appreciate your time and
effort.

Please make sure to provide enough information so that we can review
your pull
request. The Summary and Testing sections below contain guidance on what
to
include.
-->

## Summary

<!--
If this PR is related to a tracked effort, please link to the relevant
issue
here (e.g., `Closes: vortex-data#123`). Otherwise, feel free to ignore / delete
this.

In this section, please:

1. Explain the rationale for this change.
2. Summarize the changes included in this PR.

A general rule of thumb is that larger PRs should have larger summaries.
If
there are a lot of changes, please help us review the code by explaining
what
was changed and why.

If there is an issue or discussion attached, there is no need to
duplicate all
the details, but clarity is always preferred over brevity.
-->

Closes: #000

<!--
## API Changes

Uncomment this section if there are any user-facing changes.

Consider whether the change affects users in one of the following ways:

1. Breaks public APIs in some way.
2. Changes the underlying behavior of one of the engine integrations.
3. Should some documentation be updated to reflect this change?

If a public API is changed in a breaking manner, make sure to add the
appropriate label. You can run `./scripts/public-api.sh` locally to see
if there
are any public API changes (and this also runs in our CI).
-->

## Testing

<!--
Please describe how this change was tested. Here are some common
categories for
testing in Vortex:

1. Verifying existing behavior is maintained.
2. Verifying new behavior and functionality works correctly.
3. Serialization compatibility (backwards and forwards) should be
maintained or
   explicitly broken.
-->

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…ex-data#7016)

This adds a check to CI that ensures that files generated by the last
release are still readable in this PR.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Port min-max to an AggregateFn.

Note that we could push-down over dictionary arrays with the given
property (similarly for RunEnd):

```python
def is_support_invariant(agg_func) -> bool:
    """
    Returns True if this aggregate function depends only on the set of
    distinct values in its input, not on how many times each appears.

    Formally, for any multiset S:

        agg(S) == agg(set(S))

    This is the property that allows the aggregate to be evaluated purely
    over dictionary values, skipping the codes column entirely. Since a
    dictionary already contains exactly the distinct values present, no
    codes scan is needed.

    Examples:
        is_support_invariant(min)           # True
        is_support_invariant(max)           # True
        is_support_invariant(count_distinct) # True
        is_support_invariant(bool_or)       # True
        is_support_invariant(bool_and)      # True
        is_support_invariant(any_value)     # True
        is_support_invariant(median)        # False — needs frequencies
        is_support_invariant(mode)          # False — entirely about frequencies
        is_support_invariant(sum)           # False
        is_support_invariant(count)         # False
```

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary

When we compare `Scalar`s, we have to be careful that we do not use `<`
or `>` if we do not know that the scalars have different types. If they
have different types, the result will not panic or raise an error, it
will just return `false` for `>`.

This changes the `scalar_cmp` function inside the comparison execution
to use the `partial_cmp` method directly, where we map the optional
result to an error and raise it.

## Testing

Adds 2 regression tests, the first one is the one I stumbled on and the
second is a more targeted one

## Unresolved Questions

- Why is this function public?
- Are there any other places that we do scalar comparison with the
built-in `>` operators? If so, we may need to fix them.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Co-authored-by: Joe Isaacs <joe.isaacs@live.co.uk>
…nge (vortex-data#7010)

This was a little bit too complex given what we were trying to do

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary

Improves docs on `ScalarFnVTable`.

## Testing

N/A

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…ortex-data#7008)

Instead of always casting values to i256 we take the primitive value of
the return dtype and perform operations in that space

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-authored-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
NaN will cause sum to be NaN. Instead we have NaNCount already that
users can
use to handle columns with NaNs.

fix vortex-data#5152 

Signed-off-by: Robert Kruszewski <github@robertk.io>

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-authored-by: Nicholas Gates <nick@nickgates.com>
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [fsst-rs](https://redirect.github.com/spiraldb/fsst) |
workspace.dependencies | patch | `0.5.6` → `0.5.7` |

---

### Release Notes

<details>
<summary>spiraldb/fsst (fsst-rs)</summary>

###
[`v0.5.7`](https://redirect.github.com/spiraldb/fsst/blob/HEAD/CHANGELOG.md#057---2026-03-17)

[Compare
Source](https://redirect.github.com/spiraldb/fsst/compare/v0.5.6...v0.5.7)

##### Fixed

- Handle exactly-sized buffers in `compress_into`/`decompress_into`
([#&#8203;165](https://redirect.github.com/spiraldb/fsst/pull/165))

##### Other

- no more duplicate candidate generation
([#&#8203;181](https://redirect.github.com/spiraldb/fsst/pull/181))
- *(deps)* lock file maintenance
([#&#8203;180](https://redirect.github.com/spiraldb/fsst/pull/180))
- *(deps)* update swatinem/rust-cache digest to
[`e18b497`](https://redirect.github.com/spiraldb/fsst/commit/e18b497)
([#&#8203;179](https://redirect.github.com/spiraldb/fsst/pull/179))
- *(deps)* lock file maintenance
([#&#8203;178](https://redirect.github.com/spiraldb/fsst/pull/178))
- *(deps)* lock file maintenance
([#&#8203;176](https://redirect.github.com/spiraldb/fsst/pull/176))
- Remove codspeed walltime benchmark
([#&#8203;177](https://redirect.github.com/spiraldb/fsst/pull/177))
- Add more micro benchmarks
([#&#8203;171](https://redirect.github.com/spiraldb/fsst/pull/171))
- *(deps)* update marcoieni/release-plz-action digest to
[`1528104`](https://redirect.github.com/spiraldb/fsst/commit/1528104)
([#&#8203;170](https://redirect.github.com/spiraldb/fsst/pull/170))
- *(deps)* update codspeedhq/action digest to
[`281164b`](https://redirect.github.com/spiraldb/fsst/commit/281164b)
([#&#8203;169](https://redirect.github.com/spiraldb/fsst/pull/169))
- *(deps)* update actions/upload-artifact action to v7
([#&#8203;167](https://redirect.github.com/spiraldb/fsst/pull/167))
- *(deps)* lock file maintenance
([#&#8203;168](https://redirect.github.com/spiraldb/fsst/pull/168))
- *(deps)* update actions/upload-artifact action to v6
([#&#8203;160](https://redirect.github.com/spiraldb/fsst/pull/160))
- *(deps)* lock file maintenance
([#&#8203;164](https://redirect.github.com/spiraldb/fsst/pull/164))
- *(deps)* update swatinem/rust-cache digest to
[`779680d`](https://redirect.github.com/spiraldb/fsst/commit/779680d)
([#&#8203;157](https://redirect.github.com/spiraldb/fsst/pull/157))
- *(deps)* update actions/checkout digest to
[`de0fac2`](https://redirect.github.com/spiraldb/fsst/commit/de0fac2)
([#&#8203;158](https://redirect.github.com/spiraldb/fsst/pull/158))
- *(deps)* update codspeedhq/action digest to
[`2ac5728`](https://redirect.github.com/spiraldb/fsst/commit/2ac5728)
([#&#8203;162](https://redirect.github.com/spiraldb/fsst/pull/162))
- *(deps)* update marcoieni/release-plz-action digest to
[`f708778`](https://redirect.github.com/spiraldb/fsst/commit/f708778)
([#&#8203;166](https://redirect.github.com/spiraldb/fsst/pull/166))
- *(deps)* update marcoieni/release-plz-action digest to
[`52440b5`](https://redirect.github.com/spiraldb/fsst/commit/52440b5)
([#&#8203;156](https://redirect.github.com/spiraldb/fsst/pull/156))
- *(deps)* lock file maintenance
([#&#8203;161](https://redirect.github.com/spiraldb/fsst/pull/161))
- *(deps)* lock file maintenance
([#&#8203;159](https://redirect.github.com/spiraldb/fsst/pull/159))
- *(deps)* update actions/checkout action to v6
([#&#8203;154](https://redirect.github.com/spiraldb/fsst/pull/154))
- *(deps)* lock file maintenance
([#&#8203;155](https://redirect.github.com/spiraldb/fsst/pull/155))
- *(deps)* update codspeedhq/action digest to
[`346a2d8`](https://redirect.github.com/spiraldb/fsst/commit/346a2d8)
([#&#8203;152](https://redirect.github.com/spiraldb/fsst/pull/152))
- *(deps)* update actions/checkout digest to
[`93cb6ef`](https://redirect.github.com/spiraldb/fsst/commit/93cb6ef)
([#&#8203;151](https://redirect.github.com/spiraldb/fsst/pull/151))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Port IsSorted compute function over to an AggregateFn

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [fsst-rs](https://redirect.github.com/spiraldb/fsst) |
workspace.dependencies | patch | `0.5.7` → `0.5.8` |

---

### Release Notes

<details>
<summary>spiraldb/fsst (fsst-rs)</summary>

###
[`v0.5.8`](https://redirect.github.com/spiraldb/fsst/blob/HEAD/CHANGELOG.md#058---2026-03-19)

[Compare
Source](https://redirect.github.com/spiraldb/fsst/compare/v0.5.7...v0.5.8)

##### Other

- Use maintained version of fxhash
([#&#8203;182](https://redirect.github.com/spiraldb/fsst/pull/182))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [gradle](https://gradle.org)
([source](https://redirect.github.com/gradle/gradle)) | minor | `9.2.1`
→ `9.4.0` |

---

### Release Notes

<details>
<summary>gradle/gradle (gradle)</summary>

###
[`v9.4.0`](https://redirect.github.com/gradle/gradle/compare/v9.3.1...v9.4.0)

[Compare
Source](https://redirect.github.com/gradle/gradle/compare/v9.3.1...v9.4.0)

###
[`v9.3.1`](https://redirect.github.com/gradle/gradle/releases/tag/v9.3.1):
9.3.1

[Compare
Source](https://redirect.github.com/gradle/gradle/compare/v9.3.0...v9.3.1)

This is a patch release for 9.3.0. We recommend using 9.3.1 instead of
9.3.0.

The following issues were resolved:

- [Cannot find testcases from Android Screenshot Test plugin since
Gradle 9.3.0](https://redirect.github.com/gradle/gradle/issues/36320)
- [Excluding dependencies from included builds doesn't work in Gradle
9.3.0](https://redirect.github.com/gradle/gradle/issues/36331)
- [ExternalDependency and DependencyConstraint cannot be passed to
DependencyResolveDetails#useTarget](https://redirect.github.com/gradle/gradle/issues/36359)
- [Gradle 9.3.0 generate JUnit test result files with wrong
name](https://redirect.github.com/gradle/gradle/issues/36379)
- [Build cache cannot handle outputs with non-BMP characters in the
filename](https://redirect.github.com/gradle/gradle/issues/36387)
- [Emojis in test names should not break build
caching](https://redirect.github.com/gradle/gradle/issues/36395)
- [Non utf-8 c code is no longer
buildable](https://redirect.github.com/gradle/gradle/issues/36399)
- [Breaking change in 9.3.0 regarding cross-project dependency
manipulation](https://redirect.github.com/gradle/gradle/issues/36428)
- [JUnit3 tests cannot be run with Gradle
9.3.0](https://redirect.github.com/gradle/gradle/issues/36451)
- [Test.setScanForTestClasses(false) causes all junit4 tests to be
skipped](https://redirect.github.com/gradle/gradle/issues/36508)

[Read the Release
Notes](https://docs.gradle.org/9.3.1/release-notes.html)

#### Upgrade instructions

Switch your build to use Gradle 9.3.1 by updating your wrapper:

```
./gradlew wrapper --gradle-version=9.3.1 && ./gradlew wrapper
```

See the Gradle [9.x upgrade
guide](https://docs.gradle.org/9.3.1/userguide/upgrading_version_9.html)
to learn about deprecations, breaking changes and other considerations
when upgrading.

For Java, Groovy, Kotlin and Android compatibility, see the [full
compatibility
notes](https://docs.gradle.org/9.3.1/userguide/compatibility.html).

#### Reporting problems

If you find a problem with this release, please file a bug on [GitHub
Issues](https://redirect.github.com/gradle/gradle/issues) adhering to
our issue guidelines.
If you're not sure you're encountering a bug, please use the
[forum](https://discuss.gradle.org/c/help-discuss).

We hope you will build happiness with Gradle, and we look forward to
your feedback via [Twitter](https://twitter.com/gradle) or on
[GitHub](https://redirect.github.com/gradle).

###
[`v9.3.0`](https://redirect.github.com/gradle/gradle/compare/v9.2.1...v9.3.0)

[Compare
Source](https://redirect.github.com/gradle/gradle/compare/v9.2.1...v9.3.0)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…ta#7033)

This PR contains the following updates:

| Package | Change |
[Age](https://docs.renovatebot.com/merge-confidence/) |
[Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| com.diffplug.spotless | `8.1.0` → `8.4.0` |
![age](https://developer.mend.io/api/mc/badges/age/maven/com.diffplug.spotless:com.diffplug.spotless.gradle.plugin/8.4.0?slim=true)
|
![confidence](https://developer.mend.io/api/mc/badges/confidence/maven/com.diffplug.spotless:com.diffplug.spotless.gradle.plugin/8.1.0/8.4.0?slim=true)
|

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…#7034)

This PR contains the following updates:

| Package | Change |
[Age](https://docs.renovatebot.com/merge-confidence/) |
[Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| com.gradleup.shadow | `9.2.2` → `9.4.0` |
![age](https://developer.mend.io/api/mc/badges/age/maven/com.gradleup.shadow:com.gradleup.shadow.gradle.plugin/9.4.0?slim=true)
|
![confidence](https://developer.mend.io/api/mc/badges/confidence/maven/com.gradleup.shadow:com.gradleup.shadow.gradle.plugin/9.2.2/9.4.0?slim=true)
|

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…ortex-data#7036)

This PR contains the following updates:

| Package | Change |
[Age](https://docs.renovatebot.com/merge-confidence/) |
[Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
|
[com.vanniktech.maven.publish](https://redirect.github.com/vanniktech/gradle-maven-publish-plugin)
| `0.35.0` → `0.36.0` |
![age](https://developer.mend.io/api/mc/badges/age/maven/com.vanniktech.maven.publish:com.vanniktech.maven.publish.gradle.plugin/0.36.0?slim=true)
|
![confidence](https://developer.mend.io/api/mc/badges/confidence/maven/com.vanniktech.maven.publish:com.vanniktech.maven.publish.gradle.plugin/0.35.0/0.36.0?slim=true)
|

---

### Release Notes

<details>
<summary>vanniktech/gradle-maven-publish-plugin
(com.vanniktech.maven.publish)</summary>

###
[`v0.36.0`](https://redirect.github.com/vanniktech/gradle-maven-publish-plugin/blob/HEAD/CHANGELOG.md#0360--2026-01-13-)

[Compare
Source](https://redirect.github.com/vanniktech/gradle-maven-publish-plugin/compare/0.35.0...0.36.0)

**BREAKING**

- Updated minimum supported JDK, Gradle, Android Gradle Plugin and
Kotlin versions.
- Removed support for Dokka v1, it's now required to use Dokka in v2
mode.
- Mark `DirectorySignatureType` internal.

**Behavior changes**

- `validateDeployment` now has the `DeploymentValidation` enum as type
instead of being a boolean. The default
is now to just wait for the `VALIDATED` state. The previous behavior can
be achieved by setting it to `PUBLISHED`.
  `NONE` can be used for disabling the validation completely.
- When calling `configure(...)` manually to configure what to publish
and not passing `javadocJar` explicity,
  the plugin now defaults to publishing an empty javadoc jar.

**Features**

- Android projects now support using Dokka for javadoc creation, this
will happen automatically
when using the default options and the Dokka plugin is applied to the
project.
- Added consistent `JavadocJar` and `SourcesJar` options to
`configureBasedOnAppliedPlugins` and to all
applicable project types that can be passed to `configure`. The previous
`Boolean` based versions have
  been deprecated.
- When enabling Maven Central publishing through the DSL, the
`mavenCentralDeploymentValidation` and
`mavenCentralAutomaticPublishing` are used for the default values of the
2 parameters when they are not passed
explicitly. This allows to more easily override them in certain
environments.
- When isolated projects is enabled the module/project specific
`gradle.properties` files are now considered in
  the same way they are when isolated projects is disabled.

**Improvements**

- Better error message when Maven Central credentials are missing.

##### Minimum supported versions

- JDK 17
- Gradle 9.0.0
- Android Gradle Plugin 8.13.0
- Kotlin Gradle Plugin 2.2.0

##### Compatibility tested up to

- JDK 25
- Gradle 9.3.0
- Gradle 9.4.0-milestone-4
- Android Gradle Plugin 8.13.2
- Android Gradle Plugin 9.0.0
- Android Gradle Plugin 9.1.0-alpha05
- Kotlin Gradle Plugin 2.3.0
- Kotlin Gradle Plugin 2.3.20-Beta1

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [criterion](https://criterion-rs.github.io/book/index.html)
([source](https://redirect.github.com/criterion-rs/criterion.rs)) |
workspace.dependencies | minor | `0.7` → `0.8` |

---

### Release Notes

<details>
<summary>criterion-rs/criterion.rs (criterion)</summary>

###
[`v0.8.2`](https://redirect.github.com/criterion-rs/criterion.rs/blob/HEAD/CHANGELOG.md#082---2026-02-04)

[Compare
Source](https://redirect.github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.1...criterion-v0.8.2)

##### Fixed

- don't build alloca on unsupported targets

##### Other

- *(deps)* bump crate-ci/typos from 1.40.0 to 1.43.0
- Fix panic with uniform iteration durations in benchmarks
- Update Readme
- Exclude development scripts from published package

###
[`v0.8.1`](https://redirect.github.com/criterion-rs/criterion.rs/blob/HEAD/CHANGELOG.md#081---2025-12-07)

[Compare
Source](https://redirect.github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.0...criterion-v0.8.1)

##### Fixed

- Homepage link

##### Other

- *(deps)* bump crate-ci/typos from 1.23.5 to 1.40.0
- *(deps)* bump jontze/action-mdbook from 3 to 4
- *(deps)* bump actions/checkout from 4 to 6

###
[`v0.8.0`](https://redirect.github.com/criterion-rs/criterion.rs/blob/HEAD/CHANGELOG.md#080---2025-11-29)

[Compare
Source](https://redirect.github.com/criterion-rs/criterion.rs/compare/0.7.0...criterion-v0.8.0)

##### BREAKING

- Drop async-std support

##### Changed

- Bump MSRV to 1.86, stable to 1.91.1

##### Added

- Add ability to plot throughput on summary page.
- Add support for reporting throughput in elements and bytes -
`Throughput::ElementsAndBytes` allows the text summary to report
throughput in both units simultaneously.
- Add alloca-based memory layout randomisation to mitigate memory
effects on measurements.
- Add doc comment to benchmark runner in criterion\_group macro (removes
linter warnings)

##### Fixed

- Fix plotting NaN bug

##### Other

- Remove Master API Docs links temporarily while we restore the docs
publishing.

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [noodles-bgzf](https://redirect.github.com/zaeleus/noodles) |
workspace.dependencies | minor | `0.44.0` → `0.46.0` |

---

### Release Notes

<details>
<summary>zaeleus/noodles (noodles-bgzf)</summary>

###
[`v0.46.0`](https://redirect.github.com/zaeleus/noodles/compare/noodles-bgzf-0.45.0...noodles-bgzf-0.46.0)

[Compare
Source](https://redirect.github.com/zaeleus/noodles/compare/noodles-bgzf-0.45.0...noodles-bgzf-0.46.0)

###
[`v0.45.0`](https://redirect.github.com/zaeleus/noodles/compare/noodles-bgzf-0.44.0...noodles-bgzf-0.45.0)

[Compare
Source](https://redirect.github.com/zaeleus/noodles/compare/noodles-bgzf-0.44.0...noodles-bgzf-0.45.0)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
vortex-data#7035)

This PR contains the following updates:

| Package | Change |
[Age](https://docs.renovatebot.com/merge-confidence/) |
[Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| com.palantir.consistent-versions | `3.7.0` → `3.15.0` |
![age](https://developer.mend.io/api/mc/badges/age/maven/com.palantir.consistent-versions:com.palantir.consistent-versions.gradle.plugin/3.15.0?slim=true)
|
![confidence](https://developer.mend.io/api/mc/badges/confidence/maven/com.palantir.consistent-versions:com.palantir.consistent-versions.gradle.plugin/3.7.0/3.15.0?slim=true)
|

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [x] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [noodles-vcf](https://redirect.github.com/zaeleus/noodles) |
workspace.dependencies | minor | `0.82.0` → `0.86.0` |

---

### Release Notes

<details>
<summary>zaeleus/noodles (noodles-vcf)</summary>

###
[`v0.86.0`](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.85.0...noodles-vcf-0.86.0)

[Compare
Source](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.85.0...noodles-vcf-0.86.0)

###
[`v0.85.0`](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.84.0...noodles-vcf-0.85.0)

[Compare
Source](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.84.0...noodles-vcf-0.85.0)

###
[`v0.84.0`](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.83.0...noodles-vcf-0.84.0)

[Compare
Source](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.83.0...noodles-vcf-0.84.0)

###
[`v0.83.0`](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.82.0...noodles-vcf-0.83.0)

[Compare
Source](https://redirect.github.com/zaeleus/noodles/compare/noodles-vcf-0.82.0...noodles-vcf-0.83.0)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Robert Kruszewski <github@robertk.io>
…a#7047)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
|
[sketches-ddsketch](https://redirect.github.com/mheffner/rust-sketches-ddsketch)
| workspace.dependencies | minor | `0.3.0` → `0.4.0` |

---

### Release Notes

<details>
<summary>mheffner/rust-sketches-ddsketch (sketches-ddsketch)</summary>

###
[`v0.4.0`](https://redirect.github.com/mheffner/rust-sketches-ddsketch/releases/tag/v0.4.0)

[Compare
Source](https://redirect.github.com/mheffner/rust-sketches-ddsketch/compare/v0.3.1...v0.4.0)

#### What's Changed

- Java binary encoding by
[@&#8203;PSeitz-dd](https://redirect.github.com/PSeitz-dd) in
[#&#8203;20](https://redirect.github.com/mheffner/rust-sketches-ddsketch/pull/20)
- Bump for 0.4.0 by
[@&#8203;mheffner](https://redirect.github.com/mheffner) in
[#&#8203;21](https://redirect.github.com/mheffner/rust-sketches-ddsketch/pull/21)

#### New Contributors

- [@&#8203;PSeitz-dd](https://redirect.github.com/PSeitz-dd) made their
first contribution in
[#&#8203;20](https://redirect.github.com/mheffner/rust-sketches-ddsketch/pull/20)

**Full Changelog**:
<mheffner/rust-sketches-ddsketch@v0.3.1...v0.4.0>

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [sysinfo](https://redirect.github.com/GuillaumeGomez/sysinfo) |
workspace.dependencies | minor | `0.37.0` → `0.38.0` |

---

### Release Notes

<details>
<summary>GuillaumeGomez/sysinfo (sysinfo)</summary>

###
[`v0.38.4`](https://redirect.github.com/GuillaumeGomez/sysinfo/blob/HEAD/CHANGELOG.md#0384)

[Compare
Source](https://redirect.github.com/GuillaumeGomez/sysinfo/compare/v0.38.3...v0.38.4)

- macOS: Unpin `objc2-*` crates version.
- Linux: Fix computation of disk size.

###
[`v0.38.3`](https://redirect.github.com/GuillaumeGomez/sysinfo/blob/HEAD/CHANGELOG.md#0383)

[Compare
Source](https://redirect.github.com/GuillaumeGomez/sysinfo/compare/v0.38.2...v0.38.3)

- macOS: Improve accuracy of `System::available_memory` and
`System::used_memory`.
- Documentation: add missing unit for `CPU::frequency` returned value.

###
[`v0.38.2`](https://redirect.github.com/GuillaumeGomez/sysinfo/blob/HEAD/CHANGELOG.md#0382)

[Compare
Source](https://redirect.github.com/GuillaumeGomez/sysinfo/compare/v0.38.1...v0.38.2)

- Implement `Error` trait for `MacAddrFromStrError`,
`IpNetworkFromStrError` and `KillError` types.

###
[`v0.38.1`](https://redirect.github.com/GuillaumeGomez/sysinfo/blob/HEAD/CHANGELOG.md#0381)

[Compare
Source](https://redirect.github.com/GuillaumeGomez/sysinfo/compare/v0.38.0...v0.38.1)

- macOS: Put back support for versions below macOS 12.

###
[`v0.38.0`](https://redirect.github.com/GuillaumeGomez/sysinfo/blob/HEAD/CHANGELOG.md#0380)

[Compare
Source](https://redirect.github.com/GuillaumeGomez/sysinfo/compare/v0.37.2...v0.38.0)

- Add NetBSD support.
- Windows: Fix unsoundness for a function used in `Motherboard` and
`Product`.
- Linux: Improve CPU info parsing.
- Fix `serde` serialization of `MacAddr` and of `Disk::file_system`.

</details>

---

### Configuration

📅 **Schedule**: Branch creation - Between 12:00 AM and 03:59 AM, only on
Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule
defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My42Ni40IiwidXBkYXRlZEluVmVyIjoiNDMuNjYuNCIsInRhcmdldEJyYW5jaCI6ImRldmVsb3AiLCJsYWJlbHMiOlsiY2hhbmdlbG9nL2Nob3JlIl19-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
joseph-isaacs and others added 20 commits April 8, 2026 17:33
use iterative execution in vortex array

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Fix semantic merge conflict from develop

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Initial PR to move towards a session-defined buffer allocation.

1. Sets up a MemorySession (we could move this and all buffer handle
types to vortex-io and treat the i/o crate as engine integration?)
2. Defines an allocator trait, returning a WritableHostBuffer
3. Freeze the WriteableHostBuffer into a host ByteBuffer

For now, we keep the existing behavior for device buffers (i.e.
vortex-cuda constructs its own device BufferHandle inside its segment
source).

This change configures host VortexReadAt sources (ObjectStoreReadAt and
FileReadAt) to use the allocator from the session. It also demonstrates
using the allocator from within an array (chunked canonical), although
it doesn't do the plumbing work all the way into the array builders.

A follow up PR will implement a pooling allocator.

Further PRs wlll port over the execution logic to use the allocator from
the ExecutionCtx, eventually using clippy to forbid direct ByteBufferMut
access from within our (non allocator) crates.

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
## Summary

Tracking issue: vortex-data#7297

This is the first step in decomposing `TurboQuant` as 2 scalar fns on
top of a dictionary.

We would like to (soon) decompose `TurboQuant` into a `Scheme` that
roughly constructs the following array tree:

```
L2Denorm(
    norms,
    SorfTransform(
        rotation_signs,
        Dict(
            codes, (indices into centroids)
            values, (centroid book)
        ),
    ),
)
```

This is the outer layer.

## Testing

The logic is the same so existing tests suffice.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary

Tracking issue: vortex-data#7216

We have this generic RLE code from the old compressor that doesn't make
as much sense anymore.
This change hardcodes the 2 schemes (int and float) instead of using
generics that make things harder to trace.

This is a purely cosmetic change.

## API Changes

Removes olds constants and just has `FloatRLEScheme` and `IntRLEScheme`.

## Testing

<!--
Please describe how this change was tested. Here are some common
categories for
testing in Vortex:

1. Verifying existing behavior is maintained.
2. Verifying new behavior and functionality works correctly.
3. Serialization compatibility (backwards and forwards) should be
maintained or
   explicitly broken.
-->

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The cheeky thing here is that now we require a VortexSession to
serialize metadata:
1. Makes it hard to serialize arrays you didn't intend to. All arrays
must be registered at least.
2. Allows ScalarFnArray to serialize metadata using a dedicated
ArrayPlugin for that ScalarFn. So not all scalar function vtables need
to define array serde semantics, they just register an optional plugin.
This will allow us to port existing scalar arrays to scalar functions
without a wire break

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Mute TSan data race in oneshot as it's a false positive

Signed-off-by: Mikhail Kot <to@myrrc.dev>
## Summary

@gatesn pointed out to me that we disable incremental builds pretty
selectively, and reading through the CI code we have many partial
configuration that can be applied more widely.
I've attempted to unify all of them into a ci-specific build profile,
and removed some flags that we pass twice (once in CI, and another
already in `.cargo/config.toml`)

---------

Signed-off-by: Adam Gutglick <adam@spiraldb.com>
This allows us to enable deserialization and re-serialization of unknown
plugins by preserving their serialized metadata.

We cannot necessarily execute them in-memory, but it should be
sufficient to traverse the expression trees. For arrays/layouts, we
cannot construct the child trees without the vtable logic that derives
the child dtypes, therefore a foreign encoding becomes a leaf node.

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ortex-data#7359)

The in the avx2 take code correctly states that we need to fitler out
values
greater than but then it also filters out values that are equal, this
fixes the
edge case for values that are equal to the max index value

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
…gs (vortex-data#7362)

Signed-off-by: Robert Kruszewski <github@robertk.io>
…ata#7354)

## Summary

`patch_chunk` panics with index-out-of-bounds when decompressing sliced
ALP arrays with patches. The slice boundary must fall mid-chunk
(non-1024-aligned) to trigger it.

**Root cause:** `Patches::slice()` slices `chunk_offsets` at chunk
granularity but `indices`/`values` at element granularity. When a slice
ends mid-chunk, `patches_end_idx` can exceed `patches_indices.len()`.
The start index already used `saturating_sub` — the end index didn't.

**Fix:**
`.saturating_sub(offset_within_chunk).min(patches_indices.len())` on the
end index computation in `patch_chunk`, `search_index_chunked`, and
`search_index_chunked_batch`.

**AI disclosure:** Root cause analysis, fix implementation, and test
writing were done with Claude Code under my direction and review. I
identified the bug during development benchmarking at 100M rows and
verified the fix against my workload.

## Testing

- Unit test calling `patch_chunk` with inputs that OOB without the fix.
- Integration test: ALP encode 3-chunk array with patches, slice
mid-chunk, decode. Tests both start-aligned and mid-chunk-start
variants.
- All existing tests pass (2538 vortex-array, 167 vortex-alp).

---------

Signed-off-by: Abanoub Doss <abanoub.doss@gmail.com>
Co-authored-by: Abanoub Doss <abanoub.doss@gmail.com>
Use iterative execute require_child to execute its children.

We should look into a better API to access slots. I am working on this

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary

Part of roll out of Patched array. This PR adds a new environment
variable, `VORTEX_EXPERIMENTAL_PATCHED_ARRAY`.

When the env var is set

1. On read, any flat layout containing a `BitPacked` array with interior
patches will be decoded as a `Patched` array wrapping a `BitPacked` with
no interior patches
2. The default BtrBlocks compressor will write a Patched array of
BitPacked instead of BitPacked arrary with interior patches.

This allows us to do some more soak testing of the encoding before we
switch it to be the default.

## Testing

On a previous commit in this PR, I've setup the benchmarks to run with
the experimental Patched encoding enabled, the benchmark results all
seem within the margin of error. The file sizes are generally the same
but in the worst case, 2-3% larger.

---------

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
Adds a array_slot(VTable) attr macro which can be applied to a slots
struct. This gives types access to the slots for an array.

e.g.

```rust
#[array_slots(ALP)]
pub struct ALPSlots {
    /// The ALP-encoded values array.
    pub encoded: ArrayRef,
    /// The indices of exception values that could not be ALP-encoded.
    pub patch_indices: Option<ArrayRef>,
    /// The exception values that could not be ALP-encoded.
    pub patch_values: Option<ArrayRef>,
    /// Chunk offsets for the patch indices/values.
    pub patch_chunk_offsets: Option<ArrayRef>,
}
```

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Add IsNull to vortex-jni bindings

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
Maturin logs on stdout non-deterministically race with maturin printing
the wheel path (which also goes to stdout). If the wheel path is not
last, the `Pytest - Vortex` step fails.

Signed-off-by: Daniel King <dan@spiraldb.com>
## Summary

Adds compute optimizations for the `InnerProduct` and `CosineSimilarity`
changes, mostly related to when a child has already been decomposed into
normalized and norms via `L2Denorm` scalar fn array.

## Testing

Adds some more tests.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary

Fixes the instruction to run `./scripts/public-api.sh` instead of `cargo
xtask` just for compat (first commit), and also formats everything.

## Testing

N/A

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Luke Kim <80174+lukekim@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 10, 2026 01:11
@lukekim lukekim self-assigned this Apr 10, 2026
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Syncs the repo to upstream Vortex 0.68.0, updating encoding kernels to the newer ArrayView/session-based execution APIs while also modernizing benchmark tooling, docs, and CI automation around fuzzing/benchmarks/web assets.

Changes:

  • Refactors multiple encoding compute kernels (datetime-parts, bytebool, ALP/ALPRD) to new kernel/adaptor traits and execution contexts.
  • Updates benchmarks and benchmark website tooling, plus documentation and contribution/agent guidance.
  • Reworks CI workflows for fuzzing/benchmarks/web and introduces compatibility fixture generation/validation workflows.

Reviewed changes

Copilot reviewed 134 out of 1795 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
encodings/datetime-parts/src/compute/mod.rs Exposes is_constant module and updates tests to new array construction/execution APIs
encodings/datetime-parts/src/compute/mask.rs Ports masking kernel to ArrayView and new DateTimeParts API
encodings/datetime-parts/src/compute/kernel.rs Updates parent kernel adaptors (Take/Compare) for new module paths/types
encodings/datetime-parts/src/compute/is_constant.rs Replaces old kernel registration with dyn aggregate kernel implementation
encodings/datetime-parts/src/compute/filter.rs Ports filter reduce implementation to ArrayView-based API
encodings/datetime-parts/src/compute/compare.rs Ports compare kernel signatures to ArrayView and ArrayRef
encodings/datetime-parts/src/compute/cast.rs Ports cast reduce to ArrayView and updates conformance tests
encodings/datetime-parts/src/compress.rs Adjusts temporal split validity handling and refactors TryFrom for new data type
encodings/datetime-parts/src/canonical.rs Updates decode pipeline to new buffer/validity/execution APIs and tests
encodings/bytebool/src/slice.rs Ports slice reduce to ArrayView and new validity API
encodings/bytebool/src/rules.rs Updates rule adaptors and encoding type
encodings/bytebool/src/kernel.rs Updates take adaptor path and parent kernel set type
encodings/bytebool/Cargo.toml Adds vortex-mask dependency
encodings/alp/src/lib.rs Adds session initialization helper for registering encodings + ALP NaN-count kernel
encodings/alp/src/alp_rd/slice.rs Ports slice kernel to ArrayView type and new constructor API
encodings/alp/src/alp_rd/rules.rs Updates optimizer rules to new encoding type
encodings/alp/src/alp_rd/ops.rs Updates scalar access vtable to ArrayView and dtype-based float selection
encodings/alp/src/alp_rd/mod.rs Refactors RD decode pipeline; adds fast-path combining and adjusts patches ownership
encodings/alp/src/alp_rd/kernel.rs Updates parent kernels to new adaptor module paths/types
encodings/alp/src/alp_rd/compute/take.rs Ports take execution to ArrayView, ArrayRef, and ctx-aware patches
encodings/alp/src/alp_rd/compute/mod.rs Updates consistency tests to use into_array()
encodings/alp/src/alp_rd/compute/mask.rs Ports mask reduce to new scalar-fn factory API and patches ownership
encodings/alp/src/alp_rd/compute/filter.rs Ports filter kernel to ArrayView and ctx-aware patches filtering
encodings/alp/src/alp_rd/compute/cast.rs Ports cast reduce to ArrayView and ctx-aware patches handling
encodings/alp/src/alp/rules.rs Updates parent kernels/rules to new adaptor paths and encoding type
encodings/alp/src/alp/ops.rs Ports scalar access vtable to ArrayView and dtype-based ptype selection
encodings/alp/src/alp/mod.rs Exposes compute module and updates protobuf metadata test serialization
encodings/alp/src/alp/decompress.rs Requires ExecutionCtx for execution/patching and adjusts chunk helpers
encodings/alp/src/alp/compute/take.rs Ports take execution to ArrayView and ctx-aware patches
encodings/alp/src/alp/compute/slice.rs Ports slice kernel to ArrayView
encodings/alp/src/alp/compute/nan_count.rs Replaces old kernel registration with dyn aggregate NaN-count kernel
encodings/alp/src/alp/compute/mod.rs Exposes nan_count module for registration and updates tests
encodings/alp/src/alp/compute/mask.rs Ports mask reduce/kernel to ctx-aware validity masking + patch value casting
encodings/alp/src/alp/compute/filter.rs Ports filter kernel to ArrayView and ctx-aware patch filtering
encodings/alp/src/alp/compute/compare.rs Ports compare kernel to ArrayView and ArrayRef rhs
encodings/alp/src/alp/compute/cast.rs Ports cast reduce to ArrayView and simplifies unchecked ctor usage
encodings/alp/src/alp/compute/between.rs Ports between reduce to ArrayView and ArrayRef bounds
encodings/alp/src/alp/compress.rs Updates validity API usage and returns new encoding type
encodings/alp/benches/alp_compress.rs Updates benchmark setup for new RNG API + ctx-required decompress
docs/pyproject.toml Updates Sphinx minimum version
docs/project/contributing.md Expands AI assistance policy and repo agent expectations
docs/developer-guide/internals/session.md Updates session registration examples
docs/developer-guide/internals/architecture.md Fixes table formatting and updates Spark module path naming
docs/developer-guide/benchmarking.md Major expansion of benchmarking best practices and references
docs/api/python/store.rst Replaces monolithic module doc with toctree pages + from_url function
docs/api/python/store/aws.rst Adds S3 API docs page
docs/api/python/store/azure.rst Adds Azure API docs page
docs/api/python/store/config.rst Adds common config API docs page
docs/api/python/store/gcs.rst Adds GCS API docs page
docs/api/python/store/http.rst Adds HTTP store API docs page
docs/api/python/store/local.rst Adds local store API docs page
docs/api/python/store/memory.rst Adds memory store API docs page
docs/api/python/arrays.rst Updates renamed Python API type (SerializedArray)
docs/api/cpp/index.rst Updates documented minimum C++ standard
docs/api/c/dtypes.rst Removes function docs entry for clone helper
clippy.toml Disallows Result::and due to footgun risk
benchmarks/random-access-bench/src/main.rs Updates RNG API and benchmark output writer creation
benchmarks/lance-bench/src/main.rs Switches SQL runner API to BenchmarkMode + result wrapper type
benchmarks/lance-bench/src/convert.rs Uses Arc::clone for schemas/columns
benchmarks/lance-bench/src/compress.rs Uses Arc::clone for Parquet schema
benchmarks/duckdb-bench/src/main.rs Switches to BenchmarkMode instead of early-return explain path
benchmarks/duckdb-bench/src/lib.rs Adds BenchmarkQueryResult wrapper for DuckDB results
benchmarks/datafusion-bench/src/main.rs Switches to BenchmarkMode + result wrapper; refactors object store usage
benchmarks/datafusion-bench/src/lib.rs Uses explicit Arc::clone when registering object stores
benchmarks/compress-bench/src/vortex.rs Updates array stream conversion to into_array()
benchmarks/compress-bench/src/parquet.rs Uses Arc::clone for schema
benchmarks/compress-bench/src/main.rs Updates benchmark output writer creation
benchmarks/compress-bench/src/lib.rs Updates chunked array iteration APIs and RecordBatch conversion
benchmarks-website/src/styles/index.css Adds dark-mode theme variables and UI color tweaks
benchmarks-website/src/components/Sidebar.jsx Updates benchmark data download URL
benchmarks-website/src/components/Header.jsx Adds light/dark theme logo switching via <picture>
benchmarks-website/src/components/ChartContainer.jsx Refactors imports and adjusts chart typography/tooltip styling
benchmarks-website/server.js Updates data/commits URLs and normalizes one chart label
benchmarks-website/package.json Bumps Vite major version
benchmarks-website/config.js Removed legacy config module (migrated elsewhere)
benchmarks-website/utils.js Removed legacy utils module (migrated elsewhere)
benchmarks-website/zoom-sync.js Removed legacy zoom sync module (migrated elsewhere)
benchmarks-website/worker-manager.js Removed legacy worker manager module (migrated elsewhere)
benchmarks-website/data-worker.js Removed legacy data worker (migrated elsewhere)
benchmarks-website/data-shared.js Removed legacy shared grouping logic (migrated elsewhere)
benchmarks-website/data-processor.js Removed legacy processor wrapper (migrated elsewhere)
benchmarks-website-v2/index.html Removes legacy v2 site entrypoint
benchmarks-website-v2/public/site.webmanifest Removes legacy v2 webmanifest
_typos.toml Expands ignore lists and excludes additional paths
REUSE.toml Updates licensing annotations and includes JSON under Apache-2.0 override
CONTRIBUTING.md Aligns top-level AI policy with expanded docs
CLAUDE.md Renames/repoints to AGENTS.md
AGENTS.md Adds consolidated agent/development guidelines document
.github/workflows/web.yml Adds web build/lint/deploy workflow for vortex-web
.github/workflows/wasm-fuzz.yml Updates artifact upload action version
.github/workflows/typos.yml Updates typos action version
.github/workflows/sql-pr.yml Converts PR SQL benchmarks to workflow_call entrypoint
.github/workflows/run-fuzzer.yml Refactors fuzz runner inputs, corpus naming, fork-mode repro, and runner selection
.github/workflows/report-fuzz-crash.yml Adds FUZZ_NAME plumbing and updates artifact download versions
.github/workflows/release-drafter.yml Updates release-drafter action version and simplifies token usage
.github/workflows/publish.yml Updates artifact download versions and adds compat upload workflow call
.github/workflows/publish-benchmarks-website.yml Retargets publish to benchmarks-website/** and updates docker actions
.github/workflows/package.yml Updates artifact upload versions
.github/workflows/nightly-bench.yml Updates benchmark matrix and dataset bucket for S3 runs
.github/workflows/minimize_fuzz_corpus_workflow.yml Updates runner selection and cargo-fuzz installer revision
.github/workflows/fuzzer-fix-automation.yml Updates cargo-fuzz installer revision
.github/workflows/fuzz.yml Increases schedule frequency and adds additional fuzz jobs + fork parallelism
.github/workflows/fuzz-coverage.yml Updates runner selection and artifact upload version
.github/workflows/docs.yml Updates Doxygen action and Spark javadoc path; updates artifact download version
.github/workflows/compat-validation.yml Adds scheduled compat validation workflow
.github/workflows/compat-gen-upload.yml Adds compat fixture gen/upload workflow with dry-run gating
.github/workflows/close-fixed-fuzzer-issues.yml Runs nightly and updates runner selection + cargo-fuzz installer revision
.github/workflows/bench.yml Updates AWS roles/buckets, runner selection, and bench invocation wrapper
.github/workflows/bench-pr.yml Converts to workflow_call and updates runner selection + bench invocation wrapper
.github/workflows/bench-dispatch.yml Adds label-driven dispatcher to avoid phantom check suites
.github/workflows/approvals.yml Adds PR approval requirement workflow (2 approvals for bot-authored PRs)
.github/scripts/run-sql-bench.sh Adds --iterations passthrough to benchmark binaries
.github/scripts/fuzz_report/templates/new_issue.md Improves issue template formatting and reproduction instructions
.github/scripts/fuzz_report/cli.py Uses FUZZ_NAME for issue titles (fallback to FUZZ_TARGET)
.github/runs-on.yml Switches to private runner config extension
.github/actions/setup-rust/action.yml Reworks toolchain install flow and adds optional cache timestamping
.github/actions/setup-prebuild/action.yml Adds AMI-aware setup wrapper with fork fallback
.github/actions/check-rebuild/action.yml Adds helper action to verify rebuild is a no-op
.config/nextest.toml Adds nextest override priority for a specific test filter
Files not reviewed (3)
  • .idea/copyright/profiles_settings.xml: Language not supported
  • .idea/scopes/web_files.xml: Language not supported
  • benchmarks-website/package-lock.json: Language not supported
Comments suppressed due to low confidence (1)

encodings/datetime-parts/src/compress.rs:1

  • This TryFrom<TemporalArray> implementation computes days/seconds/subseconds (and validates them) but then returns Ok(DateTimePartsData {}) without carrying those computed arrays forward. If DateTimePartsData is expected to represent the encoded data, this would make the conversion silently discard the actual payload. Consider returning a DateTimePartsData value that retains the split components (or, if DateTimePartsData is intentionally a marker/ZST, remove this TryFrom impl and expose a constructor that returns the actual array type instead).
// SPDX-License-Identifier: Apache-2.0

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/run-fuzzer.yml
Comment thread .github/workflows/run-fuzzer.yml
Comment thread encodings/alp/src/alp_rd/mod.rs
Comment thread .github/actions/setup-rust/action.yml
Comment thread benchmarks-website/src/components/Header.jsx
Signed-off-by: Luke Kim <80174+lukekim@users.noreply.github.com>
@lukekim lukekim merged commit 2ec7499 into spiceai-52 Apr 10, 2026
30 of 53 checks passed
@lukekim lukekim deleted the lukim/spiceai-52-vortex-0.68.0 branch April 10, 2026 02:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.